RE: retention policy for cached content and media

[hugogameiro]

The recent PR #19232 adding a retention policy for cached content and media can significantly reduce the resources used by Mastodon. In particular, the storage needs. This is something that admins will benefit from and reduce the network's carbon footprint. So, I am happy that it has been introduced.

Still, the more I think about it, the more I believe that deleting all remote content that is older than X days from the database (set in 'Content cache retentions period') will cause unintentional consequences for the end user, as I mentioned on #19232 (comment).

If I bookmarked a post one year ago. I expect it to be in my bookmarks one year later, even if the server that originated the post is no longer online and the content cache retention period set by the admin is 6 months. The same goes for boosts, favourites, DMs or posts I replied to. If we use the date as the single factor for deletion, we break functionality that I believe is important.

Now, I understand that only deleting posts that are older than a date if they were not interacted with like it's the case of tootctl status remove can be "computationally heavy procedure that creates extra database indices". But we could explore ways to make this content filter of interacted posts less demanding.

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

That same counter could also be used for the 'Media cache retention period' for post attachment media. As some cannot be redownloaded if the server goes offline and in some cases the post without the media makes little sense or is just blank.

There could be something I am missing, or this could be way more complex than I think. But I believe some more thinking could be done before merging this PR to the stable release.

[lazyatom]

This feels like a very pragmatic suggestion -- essentially "reference counting" the remote content.

[pisscotheque]

I would be very keen to see something like this incorporated into the change as well. We do desperately need a way to purge old statuses more easily but I would be very hesitant to use it if it didn't discriminate between content that local users had interacted with and that which they hadn't. This would make it a much more useful tool.

[nemobis]

Also worth noting that on many instances the storage utilisation is dominated by profiles' media rather than posts. #9567

[Sushubh]

I got hit by this on a recently launched instance by Vivaldi Browser team. My bookmarks pointing to older posts were disappearing. They managed to locate the issue to this setting and switched it to unlimited. Hopefully, some resolution can be found to keep interactions while still saving resources for instance owners!

[ruario]

Thanks for the report by the way @Sushubh

[cham2019]

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

This is an great suggestion to make this retention policy feature more useful. Hope it gets implemented.

[saschafoerster]

I agree fully. After understanding the consequences of the retention policy, I set it to unlimited, as I would expect my bookmarks, etc. to be stable and existing, even after years. I would also like to ask to question, what about accounts that are in "monument mode" for instance, when someone died? I personally think till now, that I would like to keep the content of that account as well.

[mackuba]

One idea that comes to mind is to add a counter for each post. Every time a local user bookmarks, favourites, boosts, is mentioned on or replies to a post, the counter on that post increases by one. If a local user removes from bookmarks, favourites, boosts, or delete their reply, it decreases by one. This way, the cache deletion would filter posts older than X date and where the number on that counter would be zero.

I think this could even be much simpler - probably in >99% of cases, once a user bookmarks, favorites, boosts etc. a status, they will not take that back later. So there could be just one flag "this status was favorited/bookmarked etc." that is just set once and never reset. If a post was favorited or bookmarked once, it almost certainly still is, so we can just as well keep that post without re-checking that. The additional storage savings we could get from trying to find all those statuses that were later removed from all bookmarks etc. would be minimal, for a lot more work.

[nondeterminischtick]

Will this action lose replies from external servers as well? So it will look like 2 years ago I was just talking to myself 😄... Or will they be retained?

I can live with disappearing boosts, favourites, and bookmarks, but would hope the context tree of my posts would remain.

[nehemiagurl]

if I understand correctly, replies are lost as well. anything that isn't in-house is gone, unless it is for some reason refetched.

[nemobis]

Yes. Replies cannot be re-fetched either, because references to them are removed as well.

[ronilaukkarinen]

Just realized this from here.

This value is blank by default, so it never deletes the remote content cache.

The current Mastodon implementation makes this a dangerous setting and will indiscriminately delete remote content older than the number of days set, whether the content was interacted with or not. Meaning, that no matter if a local user bookmarked, favourited, or boosted a remote post or even if a post is a remote reply to a local post, all will be deleted once the number of days has been reached, which is irreversible.

You can read a discussion I started in the Mastodon GitHub about the consequences of enabling this setting to see if the Mastodon dev team finds an alternative implementation.
Until Mastodon releases an improvement for Content cache retention period, I suggest leaving this value blank or, if you want/need to set it, make it something like 730 days (2 years) or higher but any value set will still cause data loss.

[VladlenO]

I just lost everything by enabling this feature. They are talking about cache, but in fact this is deleting data without the ability to update the cache. Fucking drug addicts who wrote about cache.

[cooljeanius]

I've been told that an issue I noticed with posts in threads getting disassociated from one another might actually be due to this policy: https://social.treehouse.systems/@egallager/111898518131801953

[nehemiagurl]

This really needs to be solved at some point. As an instance admin I get to a point that I either need to ask people for more money just to host old external toots, ~99% of which nobody cares for by this point, or I have to enable the really darconian feature, fully knowing that I delete people's old bookmarks in the process.
Just adding a simple tick once the post has been interacted with and never changing it again would be so helpful in ways I cannot explain. I will turn this feature on in a heartbeat had this filter existed.

[arinoch]

I had this feature on until I came across this discussion. Fortunately the policy was long enough that it wasn't affecting anything and my instance is new enough that there's not much to affect, but yes absolutely please build in a check for at least 1 bookmark on the local instance before it gets nuked. I'm not so picky about boosts, but if you want people to bookmark things, those things should still be there. I turned this feature off in a hot minute after reading this, and will turn it back on just as quickly if it gets fixed.

[colinstu12]

I agree as well. This is starting to bite us more and more as previous stuff gets wiped out. I personally would prefer it be any interaction and not just bookmarks (favs, boosts, been replied to).

[Mai-Lapyst]

It should also be mentioned that it needs to be desiced if only interacted-with posts should remain, or all posts in that thread, since some user (myself included) may only interact / bookmark the top-most toot(s) but is actually interested into the complete discussion / thread to read it later on or similar.

[nehemiagurl]

this will be much harder to implement than a simple "has this been interacted with" and as such seems unlikely to be implemented.

[Mai-Lapyst]

I dont think it is so much harder. It should be considered, since otherwise people would write bots or automated systems to auto-interact with large threads just to get these toots to stay onto the server. Ofc this can still be done without this addition, but it would certainly reduce the need for such bots / tools and thus the users of these.

And sure, for a big instance with a very big thread this certainly can eat a lot of resources up qickly, but theres always the single-user and tiny instances that just want to save their favorite messages & threads of their favourite humans. It could be made an setting so a instance admin can control how many posts should be saved implicitly.

[colinstu12]

I'd say any single interaction within any thread should also flag the rest of the thread as well to be retained.
Or at the very least let this be configurable.
I hate how old threads are breaking due to this, everything in the thread would be ideally retained - even if only like the first or last or middle post was interacted with.

[arinoch]

Is there any progress on this issue? I'd prefer not to have to inevitably upgrade the server my instance runs on simply because I ran out of space.

[mackuba]

If I understand it correctly, it was implemented here: #27733, but it wasn't released in any stable version yet, scheduled for the 4.3 release I guess, whenever that comes.

[mackuba]

Ah, no, sorry, I think that's something a bit different